step1: Extend `generate-update-metadata()` to read from `/usr` by HuijingHei · Pull Request #938 · coreos/bootupd

HuijingHei · 2025-05-29T05:53:46Z

Copy Cloin's comment from #938 (comment):
Now we have:
/usr/lib/ostree-boot -> /usr/lib/bootupd/updates/EFI

With this patch, will extend
/usr/lib/efi/<component> -> /usr/lib/bootupd/updates/EFI

HuijingHei · 2025-06-05T07:03:56Z

Ready for reviewing now.

I did testing under fedora-bootc container, and upgrade grub2-efi-x64 shim-x64 to new, remove the old /usr/lib/bootupd/updates/* and run generate-update-metadata to create metadata.

Build new version according to #926 (comment):
grub2:
https://koji.fedoraproject.org/koji/taskinfo?taskID=133336624
shim:
https://koji.fedoraproject.org/koji/taskinfo?taskID=133341841

Download and prepare the repo

# cat myrepo.repo
[myrepo]
name=My Custom Repository
baseurl=http://localhost:8080/bootloader/
enabled=1
gpgcheck=0

Build patch

# make && make install-all DESTDIR=/srv/bootupd-test/out

Run testing under fedora-bootc container

podman run --privileged -it -v ./:/opt -w /opt -v ./tmp/myrepo.repo:/etc/yum.repos.d/myrepo.repo --net host quay.io/fedora/fedora-bootc:42 bash


bash-5.2# dnf upgrade -y grub2-efi-x64 shim-x64 --repo myrepo

bash-5.2# rpm -ql grub2-efi-x64 shim-x64 | grep usr/lib/efi
/usr/lib/efi/grub2/2.12-34.fc43/EFI/fedora/grubx64.efi
/usr/lib/efi/shim/15.8-4/EFI/BOOT/BOOTX64.EFI
/usr/lib/efi/shim/15.8-4/EFI/BOOT/fbx64.efi
/usr/lib/efi/shim/15.8-4/EFI/fedora/BOOTX64.CSV
/usr/lib/efi/shim/15.8-4/EFI/fedora/mmx64.efi
/usr/lib/efi/shim/15.8-4/EFI/fedora/shim.efi
/usr/lib/efi/shim/15.8-4/EFI/fedora/shimx64.efi

(dnf reinstall -y grub2-efi-x64 shim-x64 --repo myrepo)
bash-5.2# cp -rav out/usr /
bash-5.2# rm -rf /usr/lib/bootupd/updates/*

bash-5.2# /usr/bin/bootupctl backend generate-update-metadata / -vvv
[TRACE bootupd] executing cli
[DEBUG bootupd::ostreeutil] Using dbpath "--dbpath=/usr/lib/sysimage/rpm"
Generated update layout for BIOS: grub2-tools-1:2.12-34.fc43.x86_64
[DEBUG bootupd::ostreeutil] Using dbpath "--dbpath=/usr/lib/sysimage/rpm"
Generated update layout for EFI: grub2-efi-x64-1:2.12-34.fc43.x86_64,shim-x64-15.8-4.x86_64
[DEBUG bootupd::efi] Unmounting

### check the result:
bash-5.2# cat /usr/lib/bootupd/updates/BIOS.json 
{"timestamp":"2025-05-30T03:39:14Z","version":"grub2-tools-1:2.12-34.fc43.x86_64"} 
bash-5.2# cat /usr/lib/bootupd/updates/EFI.json 
{"timestamp":"2025-05-30T06:29:19Z","version":"grub2-efi-x64-1:2.12-34.fc43.x86_64,shim-x64-15.8-4.x86_64"}

cgwalters

Overall looks sane to me, a variety of comments. I don't have anything blocking.

I think we should be testing this scenario in CI, I doubt we are? I am not up to date on things, does it require us pulling in a copr?

cgwalters · 2025-06-09T17:45:08Z

src/efi.rs

        let dest_efidir = component_updatedir(sysroot_path, self);

-        if ostreebootdir.exists() {
+        // New EFI dir /usr/lib/efi


Do you know if this this specific to the new Fedora grub/shim packages? I suspect it is...we may want to somehow make this a build time or even runtime conditional so in theory it's more pluggable/controllable by others.

I'd at least factor it out into a const somewhere that explains where it came from.

Hmm actually, this topic also strongly relates to #766 right?

Thank you so much for the reviewing!

Do you know if this this specific to the new Fedora grub/shim packages? I suspect it is...we may want to somehow make this a build time or even runtime conditional so in theory it's more pluggable/controllable by others.

Yes.

I'd at least factor it out into a const somewhere that explains where it came from.

SGTM.

Hmm actually, this topic also strongly relates to #766 right?

Actually this is related to issue #926 (comment), but we can extend it support #766 too.

cgwalters · 2025-06-09T17:51:53Z

src/efi.rs

+                    Ok(acc)
+                });
+            packagesystem::query_files(sysroot_path, all_files?.into_iter())?
+        } else if ostreebootdir.exists() {


Would it be cleaner perhaps if we checked first for ostreebootdir, and migrated it to usr/lib/efi if that doesn't exist? It should be an error if both exist.

Then we get closer to thinking of the ostreebootdir one as legacy.

cgwalters · 2025-06-09T17:53:00Z

src/efi.rs

+                        })
+                        .collect::<Vec<String>>();
+
+                    Command::new("mv")


I think we have an opportunity here to make usr/lib/efi the standard path actually if it exists, maybe we could make things as simple as the equivalent of ln -sr /usr/lib/efi /usr/lib/bootupd/updates/EFI?

EDIT: Ah I see it's not that simple based on find_all_efi_dirs.

Hmmm...but actually I like the idea of that layout, what if we tried to adopt that as the standard and migrate our current EFI layout to it? (It'd break updates for older bootupd though...without having dual layouts for a while, ug)

Hmmm...but actually I like the idea of that layout, what if we tried to adopt that as the standard and migrate our current EFI layout to it? (It'd break updates for older bootupd though...without having dual layouts for a while, ug)

Yes, you are right, that will break old bootupd. The current EFI.json is like:

# cat EFI.json | jq { "timestamp": "2025-03-27T10:27:15Z", "version": "grub2-efi-x64-1:2.12-28.fc42.x86_64,shim-x64-15.8-3.x86_64" }

We only get /usr/lib/efi/grub2/2.12-34.fc43/EFI (-> grub2-2.12-34.fc43) and /usr/lib/efi/shim/15.8-4/EFI (-> shim-15.8-4) from the path without rpmdb, does this make sense? For silverblue we install both shim-ia32 and shim-x64, it only shows once, we do not care about this if we only concerns the version.

{ "timestamp": "<now>", "version": "grub2-2.12-34.fc43,shim-15.8-4" }

When we do the update, we sync all the files under /usr/lib/bootupd/updates/EFI to /boot/efi/EFI, means we only apply once, but if there are 2 or more directories, we need to sync each EFI directory, any good suggestion for this?

cgwalters · 2025-06-09T17:57:10Z

src/efi.rs


+// Find EFI dirs under usr/lib/efi
+// for exmaple: usr/lib/efi/shim/15.8-4/EFI, usr/lib/efi/grub2/2.12-34.fc42/EFI
+fn find_all_efi_dirs(sysroot_lib: &Path) -> Result<Vec<PathBuf>> {


BTW looking at this layout, I think we can now avoid invoking rpm to query the file information for these which would be a huge side benefit.

So again I think this relates to #766 in that perhaps we make this layout our new "API" for adding content in the ESP?

Actually thinking about things here...you know, it probably wouldn't be terribly hard to change what rpm-ostree does to automatically do this instead (via an opt-in). That'd require some coordination but the powerful benefit is we'd effectively automatically "backport" support for /usr/lib/efi even for older OSes which seems like it'd help us a lot here.

Want to clarify this, what we currently do in generate-update-metadata() is move files under legacy ostree /usr/lib/ostree-boot/efi/EFI to /usr/lib/bootupd/updates/EFI, then invoke rpm to query the file information (need to insert the /boot/efi/EFI to get the correct path) and create EFI.json to include the package info

What we want to change is: retrieve /usr/lib/efi and get EFI path, then add the package info from path and create EFI.json. The change might be easy.

What I am concern is for the installation and update, if there are 2 or more directories, we need to change the current logic to sync each EFI directory, instead of only /usr/lib/bootupd/updates/EFI. WDYT?

What we want to change is: retrieve /usr/lib/efi and get EFI path, then add the package info from path and create EFI.json. The change might be easy.

Yeah I think so.

What I am concern is for the installation and update, if there are 2 or more directories, we need to change the current logic to sync each EFI directory, instead of only /usr/lib/bootupd/updates/EFI. WDYT?

Yeah, I think that would make sense. However it would mean we need to bridge between the current list of files in bootupd-state.json vs the split directories.

Anyways in the short term what you're doing here (not changing the payload layout) is probably what we have to do in order to retain backwards compat (i.e. older clients can upgrade).

But after this work lands it'd probably be useful to try to start some work on making /usr/lib/efi style layout be supported; something like bootupctl backend install --format-version=2 as an opt in or so.

start some work on making /usr/lib/efi style layout be supported;

SGTM.

say-paul · 2025-06-10T08:35:17Z

src/efi.rs

+// Find EFI dirs under usr/lib/efi
+// for exmaple: usr/lib/efi/shim/15.8-4/EFI, usr/lib/efi/grub2/2.12-34.fc42/EFI
+fn find_all_efi_dirs(sysroot_lib: &Path) -> Result<Vec<PathBuf>> {
+    const LIBDIRS: &[&str] = &["grub2", "shim"];


Can we have : const LIBDIRS: &[&str] = &["grub2", "shim","."];
So that we can keep in things in /usr/lib/efi/<version>/EFI/<files> , EFI.json metadata can contain the rpm details.

Maybe can remove the limitation and scan all EFI directories like /usr/lib/efi/<name>/<version>/EFI/<files> ?

That can be one method, we would only need to find a name for putting things to esp, which is trivial.

Let's not consider files outside of a <name>/<version> directory here as we need to be able to know "where" they come from / attach they to a package/source, even if it's user specified.

is reliance on <name>/<version> to remove dependency on rpm -qf which does not work when files are copied around?

is reliance on <name>/<version> to remove dependency on rpm -qf which does not work when files are copied around?

Yes, I think it will not work. In future, will rely on that to get meta for package.

HuijingHei · 2025-06-13T08:16:55Z

Look at more about #938 (comment), to clarify:

Firstly checks /usr/lib/efi/(shim|grub)/%{version}-%{release}/EFI/ EFI path exists, then add EFI.json about the "(shim|grub)/%{version}-%{release}"
If the path /usr/lib/efi doesn’t exist, check the legacy OSTree location /usr/lib/ostree-boot/efi/EFI/, use rpm to query the files and translate them like /usr/lib/efi/(shim|grub)/%{version}-%{release}/EFI/, then add EFI.json about the "(shim|grub)/%{version}-%{release}"
For the installed /boot/efi files, use rpm to query the files and translate them to /usr/lib/efi/(firmware)/%{version}-%{release}/EFI/boot/efi/*, also append EFI.json about "(firmware)/%{version}-%{release}", like:

{
  "timestamp": "2025-03-27T10:27:59Z",
  "version": "grub2-2.12-28.fc42,shim-15.8-3,bcm2711-firmware-20250319-1.80f23d4.fc42,bcm2835-firmware-20250319-1.80f23d4.fc42,bcm283x-firmware-20250319-1.80f23d4.fc42,bcm283x-overlays-20250319-1.80f23d4.fc42"
}
bash-5.2# dnf install -y bcm2711-firmware
Package                                                   Arch           Version                                                   Repository                            Size
Installing:
 bcm2711-firmware                                         aarch64        20250319-1.80f23d4.fc42                                   fedora                            10.3 MiB
Installing dependencies:
 bcm2835-firmware                                         aarch64        20250319-1.80f23d4.fc42                                   fedora                            12.1 MiB
 bcm283x-firmware                                         aarch64        20250319-1.80f23d4.fc42                                   fedora                            73.6 KiB
 bcm283x-overlays                                         aarch64        20250319-1.80f23d4.fc42                                   fedora                             1.0 MiB
...
bash-5.2# find /boot/efi/ -type f -exec rpm -qf {} \; 2>/dev/null | sort | uniq
bcm2711-firmware-20250319-1.80f23d4.fc42.aarch64
bcm2835-firmware-20250319-1.80f23d4.fc42.aarch64
bcm283x-firmware-20250319-1.80f23d4.fc42.aarch64
bcm283x-overlays-20250319-1.80f23d4.fc42.aarch64

Consequently, need to change the install to copy each directory to the destination "/boot/efi". WDYT?

say-paul · 2025-06-13T09:50:41Z

@HuijingHei thank you for thinking this through, looks pretty exhaustive and cover most of the use-cases.
Though I would also like to point one more use cases:
Its related to the peculiarities of how rpi boot function works:
It requires the uboot binary to be in esp which means:

[root@192 ~]# rpm -ql uboot-images-armv8 | grep rpi
/usr/share/uboot/rpi_arm64
/usr/share/uboot/rpi_arm64/u-boot.bin

u-boot.bin has to moved to /boot/efi .
Suggestion:
generate-update-metadata to accept any source_location(in this case:/usr/share/uboot/rpi_arm64)
and followed by your suggest method for /boot/efi files.

HuijingHei · 2025-06-13T10:22:09Z

generate-update-metadata to accept any source_location(in this case:/usr/share/uboot/rpi_arm64)

Not sure how to point the file /usr/share/uboot/rpi_arm64/u-boot.bin precisely, or we only care about it? I see there are many u-boot.bin under different directories.

say-paul · 2025-06-13T10:53:34Z

Not sure how to point the file

I see this in cli/bootupd.rs is restricted to / for now:

 pub(crate) fn run_generate_meta(opts: GenerateOpts) -> Result<()> {
        let sysroot = opts.sysroot.as_deref().unwrap_or("/");
        if sysroot != "/" {
            anyhow::bail!("Using a non-default sysroot is not supported: {}", sysroot);
        }
        bootupd::generate_update_metadata(sysroot).context("generating metadata failed")?;
        Ok(())
     }

I see there are many u-boot.bin under different directories.

The package installs the binary for all the boards, but rpi needs it be in /boot/efi

Also want to understand what will happen if: cp /usr/share/uboot/rpi_arm64/u-boot.bin /boot/efi/u-boot.bin is run before generate-update-metadata() is executed, as rpm -qf will show "not owned by any package", will it still be moved to /usr/lib/efi/firmware?

HuijingHei · 2025-06-18T10:12:31Z

The package installs the binary for all the boards, but rpi needs it be in /boot/efi

Also want to understand what will happen if: cp /usr/share/uboot/rpi_arm64/u-boot.bin /boot/efi/u-boot.bin is run before generate-update-metadata() is executed, as rpm -qf will show "not owned by any package", will it still be moved to /usr/lib/efi/firmware?

I think probably no, as it can not find the correct path.

HuijingHei · 2025-07-22T08:02:22Z

One issue about this PR is the old timestamp 1970-01-01T00:00:00Z, which might hit #933 when upgrade.
    "EFI": {
      "meta": {
        "timestamp": "1970-01-01T00:00:00Z",
        "version": "grub2-2.12-34.fc43,shim-15.8-4"
      },

Workaround this with 1ec371a

travier · 2025-07-24T16:11:51Z

Thanks a lot for this. This is looking good from a quick look. I'll test it tomorrow.

travier · 2025-07-24T16:13:32Z

If I'm not mistaken, this is doing the same as https://gitlab.com/fedora/bootc/base-images/-/merge_requests/90 but more cleanly so I'll close that PR.

coreos#933

HuijingHei · 2025-07-25T01:56:25Z

If I'm not mistaken, this is doing the same as https://gitlab.com/fedora/bootc/base-images/-/merge_requests/90 but more cleanly so I'll close that PR.

Yes, I think so. Also add CI test and the EFI.json content is (from log):

{
  "timestamp": "2025-07-25T01:47:40.351545925Z",
  "version": "shim-15-15.el8_2,grub2-2.06-107.el9"
}

The timestamp is using now as 1ec371a to workaround #933, let me know if you have other way. Thank you so much for the kind review!

cgwalters

Looks reasonable to me!

cgwalters · 2025-07-25T14:58:06Z

.github/workflows/ci.yml

+        run: |
+          set -xeuo pipefail
+          # This will be helpful when we extend /usr
+          sudo podman run --rm -it --privileged localhost/bootupd:latest \


I've been trying harder to move load-bearing test logic out of GHA yaml and into something that's easier to run outside of GHA.

In this case this script could live in e.g. tests/move-content-to-usr.sh or something.

cgwalters · 2025-07-25T15:22:00Z

src/efi.rs

+/// Get EFI path under usr/lib/efi, eg. usr/lib/efi/<package>/<version>/EFI
+fn get_efi_path_from_usr(
+    sysroot: &Utf8Path,
+) -> Result<impl Iterator<Item = Result<Utf8PathBuf>> + use<'_>> {


OK as is but I think this might be cleaner if we returned something like a

struct EFIComponent { name: String, version: String, path: Utf8PathBuf }

or so

Nice suggestion, will update in later PR.

travier · 2025-07-25T15:59:29Z

The timestamp is using now as 1ec371a to workaround #933, let me know if you have other way. Thank you so much for the kind review!

I'm afraid that will work but not be what we want as that means that each new build will get a newer timestamp and we'll trigger an update on each boot? If I understand correctly we don't check the version but only the timestamp? will have to check as I hadn't realized that.

travier · 2025-07-25T15:59:55Z

So we should probably still query the RPM DB for now for those files to get the "real" timestamp. Or move to version comparison instead. There is an rpm crate that could help us do that: https://docs.rs/rpm/latest/rpm/struct.Evr.html

travier · 2025-07-25T17:37:17Z

And thanks a lot for working on this and sorry I'm so late with the reviews.

HuijingHei · 2025-07-30T02:23:37Z

So we should probably still query the RPM DB for now for those files to get the "real" timestamp. Or move to version comparison instead. There is an rpm crate that could help us do that: https://docs.rs/rpm/latest/rpm/struct.Evr.html

Agree to move version comparison and will not need to query RPM DB anymore. Thank you for the suggestion, will add that later.

struct Inspired by Colin's pointer: coreos#938 (comment) ``` return EFIComponent struct: struct EFIComponent { name: String, version: String, path: Utf8PathBuf } ```

Use `rpm::Evr` crate to parse and compare `version`. Inspired by Timothee's pointer: coreos#938 (comment) Fixes: coreos#933

Use `rpm::Evr` crate to parse and compare `version` when checking for update. Inspired by Timothee's pointer: coreos#938 (comment) Fixes: coreos#933

HuijingHei · 2025-08-04T07:05:03Z

The timestamp is using now as 1ec371a to workaround #933, let me know if you have other way. Thank you so much for the kind review!

I'm afraid that will work but not be what we want as that means that each new build will get a newer timestamp and we'll trigger an update on each boot?

No, if versions are equal (simple checking current == target), it will not trigger the update, this works if they are both in the same format (both from rpm DB or both from /usr/lib/efi path). If current != target, it will compare the timestamp. Agree we need to parse and compare the version (see #977)

Use `rpm::Evr` crate to parse and compare `version` when checking for update. Inspired by Timothee's pointer: coreos#938 (comment) Fixes: coreos#933

travier · 2025-08-04T12:59:49Z

The timestamp is using now as 1ec371a to workaround #933, let me know if you have other way. Thank you so much for the kind review!

I'm afraid that will work but not be what we want as that means that each new build will get a newer timestamp and we'll trigger an update on each boot?

No, if versions are equal (simple checking current == target), it will not trigger the update, this works if they are both in the same format (both from rpm DB or both from /usr/lib/efi path). If current != target, it will compare the timestamp. Agree we need to parse and compare the version (see #977)

OK, great, then we should probably make a release with this code so that we can have the changes land in GRUB & shim RPMs for F43.

HuijingHei requested review from cgwalters and travier May 29, 2025 06:23

HuijingHei force-pushed the generate-metadata-from-usr branch 2 times, most recently from 01b09e0 to be19115 Compare May 29, 2025 07:42

HuijingHei added the do-not-merge/work-in-progress label May 29, 2025

HuijingHei force-pushed the generate-metadata-from-usr branch 5 times, most recently from 54d905f to fa2e65c Compare June 2, 2025 11:57

openshift-merge-robot added the needs-rebase label Jun 2, 2025

HuijingHei force-pushed the generate-metadata-from-usr branch from fa2e65c to 2a8961f Compare June 2, 2025 12:19

openshift-merge-robot removed the needs-rebase label Jun 2, 2025

HuijingHei force-pushed the generate-metadata-from-usr branch 3 times, most recently from ae28fbc to efa3b42 Compare June 5, 2025 06:44

HuijingHei removed the do-not-merge/work-in-progress label Jun 5, 2025

cgwalters approved these changes Jun 9, 2025

View reviewed changes

cgwalters mentioned this pull request Jun 9, 2025

support dynamically extending/adding update payloads #766

Open

say-paul reviewed Jun 10, 2025

View reviewed changes

HuijingHei force-pushed the generate-metadata-from-usr branch from 4cac136 to 7241cdf Compare June 11, 2025 14:24

say-paul mentioned this pull request Jun 13, 2025

Bootc image does not boot on rpi fedora-iot/iot-distro#82

Open

openshift-merge-robot added the needs-rebase label Jun 17, 2025

say-paul marked this pull request as draft June 20, 2025 11:38

HuijingHei force-pushed the generate-metadata-from-usr branch from c9b5795 to 3c91cdd Compare July 21, 2025 09:52

HuijingHei changed the title ~~Extend generate-update-metadata() to read from /usr~~ step1: Extend generate-update-metadata() to read from /usr Jul 22, 2025

HuijingHei added 2 commits July 25, 2025 09:11

efi: change timestamp to now as workaround of

1ec371a

coreos#933

ci: add fake /usr/lib/efi to test generate-update-metadata()

3ac4491

HuijingHei force-pushed the generate-metadata-from-usr branch from 3c91cdd to 3ac4491 Compare July 25, 2025 01:40

cgwalters approved these changes Jul 25, 2025

View reviewed changes

cgwalters merged commit 3acb674 into coreos:main Jul 25, 2025
12 checks passed

HuijingHei deleted the generate-metadata-from-usr branch July 30, 2025 02:23

HuijingHei mentioned this pull request Jul 30, 2025

efi: improve get_efi_component_from_usr to return EFIComponent struct #975

Merged

HuijingHei added a commit to HuijingHei/bootupd that referenced this pull request Aug 4, 2025

efi: move to compare version instead of timestamp

9d3e5cd

Use `rpm::Evr` crate to parse and compare `version`. Inspired by Timothee's pointer: coreos#938 (comment) Fixes: coreos#933

HuijingHei mentioned this pull request Aug 4, 2025

package: move to compare version instead of timestamp #977

Merged

travier mentioned this pull request Aug 4, 2025

Support optional update payloads, notably for ARM/AArch64 firmwares #959

Open

HuijingHei mentioned this pull request Aug 11, 2025

Release v0.2.29 #981

Closed

41 tasks

Conversation

HuijingHei commented May 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HuijingHei commented Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cgwalters left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

say-paul Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

say-paul Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

HuijingHei commented Jun 13, 2025

Uh oh!

say-paul commented Jun 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HuijingHei commented Jun 13, 2025

Uh oh!

say-paul commented Jun 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HuijingHei commented Jun 18, 2025

Uh oh!

HuijingHei commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

travier commented Jul 24, 2025

Uh oh!

travier commented Jul 24, 2025

Uh oh!

HuijingHei commented Jul 25, 2025

Uh oh!

cgwalters left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

travier commented Jul 25, 2025

Uh oh!

travier commented Jul 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HuijingHei commented May 29, 2025 •

edited

Loading

HuijingHei commented Jun 5, 2025 •

edited

Loading

say-paul Jun 10, 2025 •

edited

Loading

say-paul Jun 10, 2025 •

edited

Loading

say-paul commented Jun 13, 2025 •

edited

Loading

say-paul commented Jun 13, 2025 •

edited

Loading

HuijingHei commented Jul 22, 2025 •

edited

Loading

travier commented Jul 25, 2025 •

edited

Loading

travier commented Aug 4, 2025 •

edited

Loading